Techniques for automatically tracking software errors

ABSTRACT

Techniques are provided for automatically tracking errors encountered by a software system. An occurrence of an error that affects performance of an operation being performed by a database server is detected. In response to detecting the occurrence, error information about the error is automatically recorded in a storage space within a database that is managed by the database server. The error information is automatically recorded by executing one or more computer instructions in a first code path of the database server, where the first code path is a separate code path than a second code path of the database server that performs the operation whose performance is affected by the error.

PRIORITY CLAIM; CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit and priority under 35 U.S.C. §120 as aDivisional of U.S. patent application Ser. No. 10/975,789, entitled“TECHNIQUES FOR AUTOMATIC SOFTWARE ERROR DIAGNOSTICS AND CORRECTION” andfiled by Sashikanth Chandrasekaran on Oct. 27, 2004, the entire contentsof which is hereby incorporated by reference as if fully set forthherein, and which claims the benefit of priority under 35 U.S.C. §119(e)to U.S. Provisional Patent Application Ser. No. 60/612,900, entitled“TECHNIQUES FOR AUTOMATIC SOFTWARE ERROR DIAGNOSTICS AND CORRECTION” andfiled by Sashikanth Chandrasekaran on Sep. 24, 2004, the entire contentsof which is hereby incorporated by reference as if fully set forthherein.

This application is related to U.S. patent application Ser. No. ______,(Attorney Docket No. 50277-3563), entitled “TECHNIQUES FOR AUTOMATICSOFTWARE ERROR DIAGNOSTICS” and filed by Sashikanth Chandrasekaran onthe same day herewith, the entire contents of which is herebyincorporated by reference as if fully set forth herein.

FIELD OF THE INVENTION

The present invention relates to software diagnostics and software errorcorrection. The invention relates more specifically to automatictracking and recording of software errors, to automatic setting ofdiagnostic features, and to automatic circumvention of software errors.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

A key problem in software systems is how to ensure that the softwaresystems are continuously available. A software system must mask andquickly recover from various failures, such as hardware failures andfailures due to errors in the software code itself.

Traditionally, software errors (also known as bugs) in the code of thesoftware system that affect the performance of the software system areclassified into “Bohrbugs” and “Heisenbugs”. Bohrbugs are 100 percentreproducible—i.e. if the same sequence of operations is executed by thesoftware system, the error will cause a failure again. Usually, softwaresystems have only a few Bohrbugs because most such errors are detectedduring the testing phase of the software system life-cycle. Heisenbugs,on the other hand, are highly dependant on the timing of various events.They are difficult to reproduce because even if the software systemexecutes the same sequence of operations, the timing of events duringthe underlying execution of these operations may vary in a way that theerror does not affect the performance of the system. Complexmulti-threaded software systems, such as operating systems or databasesystems, usually have many Heisenbugs.

There are some errors, however, which are not 100 percent reproduciblebut are not very rare either. If the software system executes the samesequence of operations a sufficient number of times, these softwareerrors are bound to cause a failure or affect the performance of theoperations. These errors can be classified as an intermediate type oferrors, and they are the most common type of errors affecting theperformance of the software systems. The reason for the intermediatetype errors being most common is that Bohrbugs are found during thebeta-testing of the software system. Heisenbugs are annoying, butusually do not cause significant downtime because they are usuallycorrected by restarting the software system. Moreover, some softwaresystems, such as database systems, have the capability to automaticallyrestore the software system to a consistent state after encountering anerror, thus further making Heisenbugs less likely to cause a significantdowntime. In fact, if the software system includes several instancesrunning in a cluster, it is possible that a Heisenbug encountered in oneinstance does not cause any downtime because other instances in thecluster will continue to be available.

The nature of the intermediate type of software errors suggests that themajority of these errors come from code that is newly introduced in thesoftware system. These errors are sufficiently reproducible and byadding one or more diagnostic events, and after sufficient informationis collected, a programmer can determine the cause of the error and fixit. Typically, errors of the intermediate type are easy to fix but hardto find. For this reason, many of these intermediate type errors arefixed in software patches or service packs that are released after thesoftware system has been introduced in the market. In contrast,Heisenbugs are hard to find and hard to fix because there may not be achance to collect the necessary diagnostic information. Heisenbugs mayexist in really old code, and some of these Heisenbugs may go unreportedby a user because they have caused a software system failure, or haveaffected the performance of the software system, just one time.

One currently available approach to correcting software errors of theintermediate type involves the participation of one or more softwareengineers or customer support personnel. When a user of a softwaresystem encounters such an error, the user files a Technical AssistanceRequest (TAR). A system support engineer, usually employed by the vendorof the software system, processes the TAR and determines that either (1)the error is known and has been fixed in a patch or service pack releasefor the software system, or (2) the error is not known.

The first case, where the error is known and fixed in a patch release,is obviously the simpler case, but even this case is complex anddifficult to resolve. Typically, the user has encountered other problemsand usually has filed multiple TARs. Thus, multiple software engineersor support personnel may become involved before determining which of thereported problems is the most serious and is causing the software systemto fail. After the support personnel determines which error is affectingthe software system or is causing the software system to fail, the userhas the choice of either upgrading to the existing patch, or waiting fora one-off patch fixing only the reported error to be released. Thispresents the user with a Hobson's choice because even an upgrade to apatch release is a complex task that may require days of planning.Hence, most of the time, the user has only one realistic option, and itis to apply a one-off patch that is already available or is to be madeavailable for the particular software system. Thus, resolving this caseeasily takes at least a day, during which time the user may continue toexperience software system failures.

The second case, where the user reported a previously unknown error, iseven more difficult and time-consuming to resolve. Typically, there areseveral rounds of interaction between the user that encountered thesoftware error and the software support personnel. In these rounds ofinteraction, diagnostic event settings are suggested to the user, theuser collects the information, and sends this information to thesoftware support personnel. In some cases, it takes several months forthe correct diagnostic event settings to be suggested and theinformation to be collected. There are many errors and omissions in thisprocess, both by the software support personnel and the administrator oradministrators at the user site. Educated guesswork is heavily usedduring this trial and error process. During this time, the user may havesuffered significant downtime.

Another currently available approach to resolving intermediate typesoftware errors is to use a separate standby software system that is amirror of the primary software system. The standby software system couldbe running an older stable release of the system. The users andapplications are failed over to the standby software system when theprimary system is unavailable. Since such fail-over represents asignificant change for users and applications alike, software systemsare switched over in this manner only in the case of major disasters atthe primary site or during long periods of planned unavailability (suchas during a major upgrade).

Another general approach to resolving intermediate software errors is touse N versions of the same software system where each version is writtenby a different set of software developers. This technique, calledN-version programming, is rarely feasible except for the highest-endsystems, because of the cost of developing and maintaining N separateversions of the software system.

All of the currently available approaches for tracking and recordingerrors affecting the performance of a software system usually involverecording the error information and the system or process stateinformation in Operating System trace files. These trace files are notmanaged and maintained by a centralized system, such as a databasesystem, and thus in order to track an error through the trace filesthese files must be correlated manually. Such manual correlation usuallyrequires great skill and expertise on behalf of the support engineerinvestigating the problem.

Therefore, there is clearly a need for techniques for tracking,diagnosing, and correcting or circumventing software errors thatovercome the shortfalls of the approaches described above.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram that depicts a possible software system forperforming the techniques for automatically tracking software errors,automatically diagnosing software errors, and automaticallycircumventing software errors.

FIG. 2 is a flow diagram that depicts a technique for automaticallytracking software errors.

FIG. 3A is a flow diagram that depicts a technique for automaticallydiagnosing errors encountered in a software system.

FIG. 3B is a flow diagram that depicts a technique for determining whichof a plurality of diagnostic features should be enabled.

FIG. 4 is a flow diagram that depicts a technique for automaticallycircumventing software errors encountered in a software system.

FIG. 5 is a block diagram that illustrates a computer system upon whichan embodiment of the invention may be implemented.

DETAILED DESCRIPTION

Methods and techniques for automatically tracking, diagnosing, andcircumventing errors in the code of a software system by the softwaresystem itself are described. In the following description, for thepurposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, that the present invention may be practicedwithout these specific details. In other instances, well-knownstructures and devices are shown in block diagram form in order to avoidunnecessarily obscuring the present invention. Furthermore, the methodsand techniques described herein are in no way limited to any particularembodiment or aspect of an embodiment. An example embodiment of thetechniques described herein is implemented on a database system. Thisexample embodiment is described for illustrative purposes only, and thetechniques described herein can be implemented by any software system.

Software Error Identification and Categorization

In order for a software system to be able to diagnose and correct asoftware error within its own code, the software system must be able touniquely identify and categorize the error with respect to the code pathof the software system where the error occurs. Depending on the type offailure that a software error causes, different software systems may usedifferent error identification and categorization schemes. For thepurposes of illustration only, described below are three categories andidentification schemes of software errors that can be used to implementthe techniques described herein.

(a) Errors causing unhandled exceptions raised by the software system.The software system raises an internal exception when it detects aninternal inconsistency in data structures or in the programming logic.These exceptions are sometimes called “assertion failures.” A uniquenumber or unique string can be used to identify a software error causingsuch an exception.

(b) Errors causing unhandled exceptions raised by the Operating System(OS) because of faulty execution in the software system code. When thesoftware system is not an Operating System, such unhandled exceptionscan be a NULL pointer dereference (segmentation violation) ordivide-by-zero arithmetic exception. In this case, the call-stack tracealong with the OS exception code can be used to uniquely identify asoftware error causing such exception.

(c) Errors causing a hang or a perceived hang. As far as a user oranother software application is concerned, a request to the softwaresystem is hanging or is deadlocked if the user or the other applicationdoes not receive a response from the software system within the expectedtime. In this situation, a software system process or thread is usuallywaiting for a certain event to occur. The software error (or perceivederror) causing a hang can be described using an event name and may befurther qualified by event identifiers. The event information and thecall stack of the waiting process can determine the location and thecondition of the error that is causing the hang. If sucherror-identifying information does not point directly to the root causeof the problem, it may be because, for example, the event has notoccurred because the event itself is waiting on another event. In suchsituations, it may be necessary to consider, in succession, the eventparameters and call stacks of all waiting processes to identify theerror.

System Overview

According to an embodiment, a software system can include components toperform the following techniques for:

(1) Automatically tracking errors encountered by the software system;

(2) Automatically diagnosing software errors; and

(3) Automatically circumventing errors encountered by the softwaresystem.

In this embodiment, a database server is entrusted with the tasks ofautomatically recording errors and managing error trace information. Aseparate storage space within a database is provided for storing errorinformation, and the database server allows querying the errorinformation using standard database queries. The software system thenautomatically determines whether any diagnostic features should beturned on in response to events occurring in the system. The softwaresystem determines that certain diagnostic features should be turned onbased on logged information about errors that have occurred in thesystem. After turning on the selected diagnostic features, the softwaresystem can automatically log error information upon subsequentoccurrences of the errors.

When certain preset conditions have been satisfied, such as, forexample, a particular error occurring a predetermined number of times inresponse to requests to perform a particular operation, the softwaresystem automatically determines that the execution code path forexecuting the operation contains a software error. The software systemthen automatically switches the execution code path for the particularoperation from the current code path to a different code path, where thedifferent code path is selected from a predetermined plurality of codepaths that can execute the particular operation. The particularoperation can be any operation that the database server can execute,including but not limited to, an operation executed in response torequests from a user, requests from the OS, or requests from processesinternal to the database server.

In different embodiments, the software system implementing thetechniques described herein can be any software application running on acomputer, including but not limited to, an Operating System, an e-mailserver, a Web server, a middle-tier application server, a firewallserver, and a client-side software application. Furthermore, a softwaresystem need not perform all three of the techniques described above. Indifferent embodiments, a software system can perform any of the threetechniques separately, can perform any combination of the threetechniques, or can perform any of the three techniques in combinationwith other techniques that may require human intervention.

FIG. 1 is a block diagram that depicts a possible software system forimplementing the techniques described herein.

FIG. 1 includes a software system 100 communicatively coupled to adatabase server 112. The software system 100 comprises an Error-TrackingComponent 110, a Diagnostic Feature Component 120, and a Path-SwitchingComponent 130. The Diagnostic Feature Component includes an ErrorDiagnostic Engine 122 that maintains and makes use of a diagnosticfeature index 124. The Path-Switching Component 130 includes aPath-Switching Engine 132 that maintains and makes use of an index ofalternate code paths 134.

The Error-Tracking Component 110, the Diagnostic Feature Component 120and the Path-Switching Component 130 are software components and may beimplemented in a variety of ways, including but not limited to, one ormore libraries of functions, one or more plug-in modules, and one ormore dynamic link-libraries. In execution, the Error-Tracking Component110, the Diagnostic Feature Component 120 and the Path-SwitchingComponent 130 can be set up in a variety of ways, including but notlimited to, one or more separate processes within the address space ofthe software system 100, one or more threads of the same process withinthe software system 100, one or more daemons within the software system100, and one or more services running outside the address space of thesoftware system 100.

Similarly, the Error Diagnostic Engine 122 and the Path-Switching Engine132 can be implemented in a variety of ways, including but not limitedto, one or more libraries of functions, one or more object-orientedclasses, and one or more dynamic link libraries. In execution, the ErrorDiagnostic Engine 122 and the Path-Switching Engine 132 can also be setup in a variety of ways, including but not limited to, one or moreseparate processes within the address space of the software system 100,and one or more threads of the same process within the software system100. The diagnostic feature index 124 and the alternate code path index134 can be implemented as any known data structure, and can bemaintained within the software system 100 or in an external repository.

The database server 112 is operatively coupled to, and manages, one ormore databases, such as database 114. The database 114 comprises adatabase storage space 116 that is used to store information abouterrors detected in the software system 110.

Automatically Tracking Software Errors

1. Storing Error Information in a Database Storage Space

When the occurrence of an error affects the performance of a softwaresystem, a database server is entrusted with the tasks of automaticallyrecording error information, and with saving and managing error traceinformation. A separate storage area is allocated to save the errorinformation, which can be in a compressed format. In an embodiment, asoftware system detects the occurrence of an error that affects theperformance of an operation by the software system. In response todetecting the occurrence of an error, the software system automaticallyrecords error information about the error in a storage space within adatabase that is managed by a database server. In a differentembodiment, the software system is a database server, and the occurrenceof an error is detected by the database server itself.

In an embodiment, the database server allocates the error informationstorage space based on configuration information received from a user,where the configuration information specifies the location and size ofthe storage space. Preferably, the storage space is organized as aseparate database. In an embodiment, the storage space is organized as acircular buffer. The database server that is entrusted with keepingtrack of errors determines whether the storage space is filled up, andif it is, then the database server automatically records the errorinformation about the current error over the oldest error informationexisting in the storage space. In one embodiment, the database serverstores error information about every error that occurs within thedatabase server; in a different embodiment, the database server storeserror information only about errors that have not been encounteredbefore, and does not store error information about errors for whicherror information has already been stored within a preset precedinginterval of time.

2. Storing and Retrieving Error Information

The database server entrusted with automatic record-keeping of errorinformation can further determine the type of the error it hasencountered, and based on the type of error, it can select what errorinformation to store in the storage space. Depending on the type andcategory of the error, the database server may store, for example, thevalue of a parameter that is maintained by the database server itself,the value of a parameter maintained by the OS, or the value of aparameter associated with a network connection. In general, the errorinformation will vary depending on the type of the error. Specificexamples of such parameters include, but are not limited to, the numberof users connected to the database server at the time of the error, thememory usage at the time of the error, the operating system load at thetime of the error, the number of network connections at the time of theerror with their respective network protocols, the number ofretransmitted packets by the network subsystem at the time of the error,and the number of dropped packets by the network subsystem at the timeof the error.

Automatically recording error information can be performed by executingone or more computer instructions in a code path that is separate fromthe code path that is used for storing user data within a database.Preferably, this separate code path is simple and thoroughly tested inorder to virtually eliminate the possibility that software errors withinthis separate code path will affect the performance of an operation bythe database server. This separate code path can be executed in a loggerdaemon process that is the last process to terminate after all otherdatabase server processes have unexpectedly terminated in order toensure that any and all generated errors will be recorded. In adifferent embodiment, the separate code path can be dedicated only toautomatic recording of error information.

In one embodiment, the database server can receive a database commandthat requests retrieval of error information from the storage space. Thedatabase command can be issued from a user or from a softwareapplication. The database server then executes the database command toretrieve the error information from the storage space and returns theerror information to the requester. The database command can be anycommand that can be performed by the database server, including but notlimited to, a Structured Query Language (SQL) command.

For example, the error information in the storage space may includemeta-data about the errors that are recorded. Using a SQL command, auser can query the meta-data to determine all errors generated in thelast 24 hours. The database command can also be a command to export toregular operation system files the error information that is saved inthe database storage space. This will allow a user to use any OSutilities, such as the UNIX grep utility, to scan the error information,and to use any popular editors, such as the UNIX vi editor, to view theerror information.

3. Functional Overview

FIG. 2 depicts a technique for automatically tracking errors that occurin a software system. In this embodiment, in step 200, the softwaresystem automatically detects the occurrence of an error that affects anoperation performed by the software system. If the error has notpreviously being detected by the software system within a presetinterval of time, then, in step 205, the software system automaticallydetermines that information about the error must be recorded in adatabase storage space. In step 215, the relevant error information isautomatically recorded in a storage space in a database that is managedby a database server. If the error has previously been detected within apreset interval of time, the software system decides in step 205 thatinformation about the error need not be recorded. The software systemthen, in step 210, disregards the occurrence of the error, and continuesto automatically detect the occurrences of other errors.

Automatically Diagnosing Software Errors

1. Determining Whether Diagnostic Features Should Be Set

In one embodiment, a software system automatically determines whethersome conditions associated with the software system warrant the use ofany diagnostic features that are not currently enabled. The softwaresystem can perform this determination in response to detecting theoccurrence of an error in the system. The conditions that warrant theuse of a diagnostic feature include, but are not limited to, theseverity of an encountered error, the number of times a particular erroris encountered within a preset time interval, and a particular eventoccurring concurrently with the error. A specific example of conditionswarranting use of a diagnostic feature would be the same error occurringat approximately the same time of the day for three consecutive days.

In another embodiment, the software system logs information about errorsas they occur in the software system, and then uses information abouterrors that have previously occurred to automatically determine whetherconditions warranting use of any diagnostic features exist. The softwaresystem can log the information in a log trace file, or can log theinformation in a database storage space managed by a database server, aspreviously described. For example, the software system can loginformation about aborted transactions on a recurring basis. Once thenumber of aborted transactions exceeds a predetermined threshold, thesoftware system looks at all the aborted transaction error informationto determine whether the majority of the aborted transactions originatedfrom the same user or the same user connection. If the that is in factthe case, the software system then automatically determines thatadditional diagnostic features must be turned on to monitor the specificuser or user connection that initiated the majority of the abortedtransactions.

2. Enabling the Selected Diagnostic Features

If the software system determines that conditions warranting the use ofdiagnostic features exist, then the software system determines whichfeature or features of a plurality of diagnostic features should beenabled. Thereafter, the software system automatically enables theselected diagnostic features.

Once the selected diagnostic features have been turned on, the softwaresystem can automatically record the diagnostic information as determinedby the diagnostic features. If the software system automatically enabledone or more diagnostic features in response to the detection of anerror, the software system can then gather the required errorinformation during any subsequent occurrences of the same error byautomatically recording the error information, as described above. Theerror information, as described above, can include not only informationabout the error itself, but also any information that can be useful indebugging the problem.

Once the diagnostic features have been enabled, the software system maybe configured to detect whether conditions for disabling the featureshave been satisfied. If the software system determines that existingconditions warrant disabling the selected diagnostic features, then thesoftware system can automatically disable these features. The conditionswarranting disabling one or more diagnostic features can simply be adetermination that enough error information has been collected because,for example, a preset number of occurrences of a particular error havebeen recorded.

3. Keeping an Index of the Available Diagnostic Features

The software system maintains an index of available diagnostic featureswhere the index has a key based on information about errors. The indexentries indicate which diagnostic feature or features need to be enabledin response to error information satisfying the conditions warrantingthe use of diagnostic features.

The information in the index, along with the index key values and theindex entries can be kept in a repository external to the softwaresystem. Keeping the index information in an external repository allowsfor more flexibility since it allows a user or administrator to changefrom time to time the diagnostic features that must be automaticallyturned on in response to encountering the occurrence of a particularerror. The information in the index can also be hard-coded, i.e. theinformation contained in the index may be dictated by informationspecified in the source code that was compiled to create the executablecode of the software system. If the index is hard-coded within thesoftware system, the error information on which the index key values arebased can be kept consistent with any modifications that are made to thecode of the software system from one release to another. For example, ifa new diagnostic feature is added to the code of the software system,then corresponding index entries can be added to the index, and can beassociated with one or more index key values, thus keeping diagnosticfeatures consistent with the changes to the software system code.

The index may be implemented in any one of a variety of ways, including,but not limited to, a relational table, a hash table, an object-orientedclass where the class properties are the index entries and the classmethods are used to search the index, and a hierarchy where the indexkeys are arranged in a hierarchical order. In response to the occurrenceof a particular error, the software system performs a look-up in theindex based on error information, to locate one or more index entriesthat indicate which diagnostic feature or features should be enabled. Indifferent embodiments, the index can maintain an entry for every uniqueerror, or the index can maintain a unique entry only for every uniquesoftware component that may be the source of an error.

4. Looking Up Diagnostic Features in a Hierarchical Index

In an embodiment, the index is organized as a hierarchical index, andthe index key includes a plurality of key-parts. The number of thekey-parts in the index key is based on the information that is recordedfor a particular error. For example, for an error that causes anunhandled exception, the index key can include a key-part correspondingto the name of the software component where the error occurs, a key-partcorresponding to the name of the particular module where the erroroccurs, and a key part corresponding to the specific error number. Foran error that causes a hang or a perceived hang, the index key caninclude a key-part corresponding to the event name, and one or morekey-parts corresponding to particular event parameters.

In this embodiment, a hierarchical search can be used to determine theclosest match between the hierarchical key value for the error thatoccurred and the hierarchical key values for the entries within theindex. Initially, a set of search values is established to include allthe error information values associated with all the key-parts of theindex key. The look-up is performed by progressively relaxing the searchthrough eliminating, from this search set, values associated withkey-parts until: (1) a match is found between the set of search valuesremaining and the values of the corresponding index key-parts, or (2)there are no values left in the set of search values. If a match isfound, then the index entry associated with the matched index key entry(as determined by the key-part values) determines which diagnosticfeature or features need to be enabled.

For example, an error causing some failure in, or affecting theperformance of, the software system can be identified as“cache.io.2103”. In this example, the index key will include threekey-parts: a key-part corresponding to the name of the softwarecomponent where the error occurs, a key-part corresponding to the nameof the particular module where the error occurs, and a key partcorresponding to the specific error number. Thus, an index key entry forthis error, if it exists, will include “cache” as the softwarecomponent, “io” as the module, and “2103” as the specific error number.The look-up of which diagnostic feature or features need to be enabledin response to the occurrence of this error will include the followingsteps:

-   -   initially establishing the set of search values to be {“cache”,        “io”, “2103”};    -   (a) attempting to find a matching index key entry, based on all        values included in the set (for example, in the first iteration        of this step, this means to find an index key entry where the        software component key-part value matches “cache”, the module        key-part value matches “io”, and the specific error key-part        value matches “2103”; in the second iteration of this step, this        means to find an index key entry where the software component        key-part value matches “cache”, and the module key-part value        matches “io”; and in the third iteration of this step, this        means to find an index key entry where the software component        key-part value matches “cache”);    -   (b) if a matching index key entry is found, then determining        which diagnostic feature or features need to be enabled based on        the index entry corresponding to the matched index key entry;    -   (c) if a matching index key entry is not found, then removing a        value from the set of search values (for example, in the first        iteration of this step, this means to remove “2103”; in the        second iteration of this step, this means to remove “io”; and in        the third iteration of this step, this means to remove “cache”);        and    -   repeating steps (a), (b), and (c) until a matching index key        entry is found, or until there are no values left in the set of        search values.

In other words, the software system would first check if there is anindex key entry matching to “cache.io.2103”, i.e. whether any diagnosticfeature or features need to be enabled for the specific error “2103”. Ifsuch index key entry is found, the software system will enable thediagnostic feature or features for the specific error “2103”. If no suchentry exists in the index key, the software system would check if thereis an index key entry matching to “cache.io”. If such index key entry isfound, the software system will enable the diagnostic feature orfeatures for the “io” module in the “cache” software component. If nosuch entry exists in the index key, the software system would check ifthere is an index key entry matching to “cache”. If such index key entryis found, the software system will enable the diagnostic feature orfeatures for the “cache” software component. If no such entry exists inthe index key, the software system will determine that no diagnosticfeature needs to be enabled in response to the occurrence of the erroridentified by “cache.io.2103”.

The key-parts of the index key can be ordered in a particular order. Forexample, in the embodiment described above the highest-ordered key-partcorresponds to general components of the software system, and thelowest-ordered component corresponds to specific errors that can beencountered in the software system. The step of removing a value fromthe set of search values is performed by removing the value associatedwith the lowest ordered key-part that has not yet been removed from thesearch set. However, there is no particular requirement that thekey-parts of an index be ordered in this manner. In differentembodiments, the index key-parts can be arranged differently on thebasis of different criteria, and the traversing of the index can beperformed according to the arrangement and the ordering criteria.

5. Functional Overview

FIG. 3A combined with FIG. 3B depict a technique for automaticallydiagnosing errors in a software system.

In step 300, the software system automatically determines whetherconditions associated with the system warrant the use of any diagnosticfeatures that are not currently enabled. If conditions that warrant theuse of one or more diagnostic features do not exist, then, in step 305,the software system automatically determines that it does not need toautomatically enable any additional diagnostic features; the softwaresystem then proceeds to step 300 to continue monitoring whether anyadditional diagnostic features should be enabled. If conditions thatwarrant the use of one or more diagnostic features do exist, then, instep 305, the software system automatically determines that one or modediagnostic features must be enabled. In step 310, the software systemautomatically determines which diagnostic features, of a plurality ofdiagnostic features, should be enabled, and in step 330 the softwaresystem automatically enables the diagnostic features selected in step310.

FIG. 3B is a blown-up flow diagram that depicts a technique forperforming step 310. To determine which of a plurality of diagnosticfeatures should be enabled, in step 312 the software system performs alook-up in an index that has a key, where the index key has a pluralityof key-parts and is based on information about errors. The softwaresystem performs the look-up to find an index entry indicating whichdiagnostic features should be enabled in response to the occurrence ofan error in the software system. In step 314, the software systemestablishes an initial set of search values to include valuescorresponding to all key-parts of the index key. In step 316 thesoftware system attempts to find a match between all values in the setof search values and the values of the corresponding index key-parts ofthe index key. If in step 316 a match is found, then in step 318 thesoftware system retrieves the index entry associated with matched indexkey, and in step 320 the software system selects which diagnosticfeature or features to enable based on the index entry. The softwaresystem then proceeds to step 330 (shown in FIG. 3A) to enable theseselected diagnostics features.

An example of a useful diagnostic feature that can be enabled in thismanner is one where the software system records trace information tohelp determine the code path that is being executed. This may includerecording important functions and their parameters, printing messageswhen infrequent conditions are encountered in program logic, or printingdumps of important data structures at various points in the code. Otherdiagnostic features in a software system can also be enabled in themanner described above, including, but not limited to, CPU utilizationfor one or more processors, memory utilization of logical and physicalmemory, network throughput, disk Input/Output (I/O) utilization, numberof user connections established by the software system within aninterval of time, number of user connections dropped by the softwaresystem within an interval of time, committed transactions per second,event message log utilization, error log utilization, number of openfile descriptors, number of unhandled exceptions occurring within thesystem, number of locks to resources granted by the software system, andnumber of deadlocks detected within the system.

If in step 316 a match is not found, then in step 322 the softwaresystem removes a value from the set of search values in order to makethe search less restrictive. In step 324 the software system checks todetermine whether there are any values left in the set of search values.If in step 324 the software system determines that there are one or morevalues left in the set of search values, the software system thenproceeds to again perform step 316. If in step 324 the software systemdetermines that there are no more values left in the set of searchvalues, then in step 326 the software system automatically determinesthat no diagnostic features should be enabled. The software system thenproceeds to step 300 (shown in FIG. 3A) to continue monitoring whetherany additional diagnostic features should be enabled.

Automatically Circumventing Errors in a Software System

1. Overview

When an error is encountered by a software system, the software systemfirst keeps a persistent record of the error information as describedabove in the section “Automatic record-keeping of errors”. If diagnosticfeatures related to a particular error are available, the softwaresystem automatically enables these diagnostic features as describedabove in the section “Automatic Setting of Diagnostic Features”, andproceeds to collect more error information upon any subsequentoccurrence of the error. While the two techniques described above will,by themselves, provide for a faster resolution of a problem caused by anerror in code of the software system, the techniques cannot bythemselves prevent any subsequent occurrence of the error in thesoftware system. This section describes a technique where the softwaresystem automatically switches the execution of an operation from onecode path to one of a plurality of alternate code paths, if they exist,when certain conditions are satisfied in the software system.

The code of a software system is usually replete with alternate codepaths that can execute the same operation. This situation is notanalogous to redundancy in hardware because in hardware the redundantparts are usually identical. In software, however, the alternate codepaths may have different algorithms, different timing sequences, anddifferent performance characteristics. Alternate code paths in softwaresystems evolve not necessarily for providing redundancy, but as a resultof optimizations developed in newer releases.

Methods and techniques for automatically circumventing errors in asoftware system by switching to alternate code paths are describedherein. A plurality of distinct code paths for performing a particularoperation is included in the code of the software system. Initially, oneof these code paths is established as the execution code path for theparticular operation, and the software system executes this executioncode path in response to requests to perform the particular operation.The software system then automatically determines whether path-switchingconditions have been satisfied. If path-switching conditions have beensatisfied, then the software system automatically establishes adifferent code path from the plurality of distinct code paths, as thecode path that is executed in response to any requests to perform theparticular operation.

A uniform mechanism is established to facilitate the automatic switchingof execution code paths. In one embodiment, branch points in thesoftware system code are used to identify the different code paths forperforming a particular operation. The branch points partition thesoftware system code into distinct call graphs. Each branch point isassociated with a separate parameter. The parameter can be declared inthe software system code as global or local variable, and can hold avalue of any available data type, including but not limited to, anumeric value, a string value, or a date. The value of the parameterindicates which code path, of the available code paths, must be executedby the software system in response to a request to perform a particularoperation. The software system can then facilitate a switch-over to adifferent code path by simply changing the value of the parameter. In adifferent embodiment, the uniform mechanism for switching is representedby an ordered list of parameters. Each parameter is associated with aseparate code path. The order of the parameters in the list determinesthe order in which the software system executes the code paths specifiedby the parameters in response to a request to perform the particularoperation. The software system switches code paths by replacing aparameter in the list with a parameter associated with a different codepath.

The particular operation described herein can be any operation that canbe performed by the software system, including but not limited to, anoperation performed in response to requests from a user, in response torequests from another software application, and in response to requestsfrom a process or thread within the software system. The software systemcan be any system running on a computer, including but not limited to,an Operating System, a database server or system, an e-mail server, amiddle-tier application server, a Web server, and any client-sidesoftware application.

2. Exploring Alternate Code Paths

In an embodiment, the software system automatically determines whetherpath-switching conditions have been satisfied by automatically recordinginformation about one or more errors in response to the occurrence ofthese errors when the software system performs a particular operation.The software system then determines whether the occurrence of the one ormore errors exceeds preset threshold criteria. An example of suchcriteria can simply be that a particular error has occurred more than Ntimes in the software system. The threshold criteria, however, is notlimited to just counting the number of error occurrences and can dependon the nature of the error, and can include any criteria which are usedto evaluate the performance of the software system. Examples of suchcriteria include, but are not limited to, Central Processing Unit (CPU)utilization, memory utilization, Input/Output throughput, and networkthroughput. If the software system determines that the preset thresholdcriteria have been exceeded, the software system then automaticallydetermines that path-switching conditions have been satisfied, andproceeds to switch-over the execution of the particular operation to adifferent code path.

In an embodiment, the software system operates in two path-switchingmodes. In the first mode, the software system executes a predeterminedcode path (the execution code path) in response to a request to performa particular operation. In this mode, the software system uses a firstset of path-switching conditions to determine whether a switch-over to adifferent code path for performing the particular operation iswarranted. However, once the first set of path-switching conditions hasbeen satisfied and a different code path has been established to executethe particular operation, the software system automatically switches tothe second mode. In the second mode, the software system uses a secondset of path-switching conditions to determine whether to continueexecuting the new different code path in response to requests to performthe particular operation. The path-switching conditions in the secondset are generally easier to satisfy than the path-switching conditionsin the first set. For example, the second set of path-switchingconditions may be satisfied if a single error is encountered during theexecution of the new different code path. If the second set ofpath-switching conditions has been satisfied, the software system thenswitches execution to a third code path that is different from the codepaths that have been explored so far. If a predetermined amount of timepasses after the software system has switched to the second mode withoutthe second set of path-switching conditions been satisfied, then thesoftware system switches back to the first mode where it uses the firstset of path-switching conditions, but continues to use the new differentcode path to perform the particular operation.

In a different embodiment, the software system uses the samepath-switching conditions in the first and the second mode. Whether toselect the same or different path-switching conditions depends on thenature of the error. If the encountered error is in a critical code pathof the software system, then the same relatively less stringentpath-switching conditions will be used to facilitate a faster search fora stable error-free path. If the encountered error is not so criticaland if its occurrence can be tolerated, then, in order to avoidexcessive code path switching, the first set of path-switchingconditions may be more stringent (for example, by requiring that theerror is encountered multiple times before switching), while the secondset of path-switching conditions may be less stringent (for example, byrequiring that the code path is switched when any error is encountered).

3. Traversing an Index of Alternate Code Paths

In an embodiment, the software system maintains a code path index ofavailable code paths for performing particular operations. The code-pathindex is similar to the index, described above, for identifying whichdiagnostic features to enable. The code path index has a key based oninformation about errors. The index entries indicate which code paths,of a plurality of distinct code paths for performing the particularoperation, can be executed in response to the path-switching conditionsbeing satisfied.

The information in the code path index, along with the index key valuesand the index entries, can be kept in a repository external to thesoftware system. The information in the index can also be hard-coded,i.e. the information contained in the index is dictated by informationspecified in the source code that was compiled to create the executablecode of the software system.

The code path index may be implemented in any of a variety of ways,including, but not limited to, a relational table, a hash table, anobject-oriented class where the properties of the class are the indexentries and the class methods are used to search the index entries, anda hierarchy with a hierarchically-arranged index keys. The softwaresystem performs a look-up in the index to locate one or more indexentries that indicate which code path of the plurality of available codepaths should be selected as the code path to execute the particularoperation. In different embodiments, the index can maintain an entry forevery unique error, or the index can maintain a unique entry only forevery unique software component that contains the error.

In an embodiment, the code path index is organized similarly to theindex of diagnostic features described above in the “Automatic settingof diagnostic features” section. The code path index is organized as ahierarchical index, and the index key includes a plurality of key-parts.The number of the key-parts in the index key is based on the informationthat is recorded for a particular error. For example, the index key caninclude a key-part corresponding to the name of the software componentwhere an error occurs, a key-part corresponding to the name of theparticular module where the error occurs, and a key part correspondingto the specific error number.

In this embodiment, a hierarchical search similar to the searchdescribed above with respect to the index of diagnostic features isused. The hierarchical search determines the closest match betweeninformation about a particular error and the value of an index key.Initially, a set of search values is established to include all thevalues associated with all the key-parts of the index key. The look-upis performed by progressively relaxing the search through eliminatingfrom this search set values associated with key-parts until: (1) a matchis found between the search values remaining in the set and the valuesof the corresponding index key-parts, or (2) there are no values left inthe set of search values. If a match is found, then the index entryassociated with the matched index key entry (as determined by thekey-part values) determines which code path from the plurality ofdistinct code paths available is to be selected as the execution codepath for the particular operation. In other words, if all alternate codepaths, indicated in a level of the index, for performing the particularoperation have been tried and an error is still encountered by thesoftware system, the software system moves to the next higher level inthe hierarchy and explores the alternate paths at that level.

For example, an error causing some failure in, or affecting theperformance of, the software system can be identified as“cache.io.2103”. In this example, the index key will include threekey-parts: a key-part corresponding to the name of the softwarecomponent where the error occurs, a key-part corresponding to the nameof the particular module where the error occurs, and a key partcorresponding to the specific error number. Thus, an index key entry forthis error, if it exists, will include “cache” as the softwarecomponent, “io” as the module, and “2103” as the specific error number.

In other words, the software system would first check if there is anindex key entry matching to “cache.io.2103”, i.e. whether there are anyalternate code paths for performing the particular operation that can beexecuted in response to encountering the specific error “2103”. If nosuch entry exists in the index key, the software system would check ifthere is an index key entry matching to “cache.io”. If such index keyentry is found, the software system will explore any alternate codepaths in the “cache.io” module. If the software system explores allalternative code paths in the “cache.io” component and still fails tocircumvent the “2103” error, the software system would check if there isan index key entry matching to “cache”. If such entry is found, thesoftware system will then explore all alternative code paths for the“cache” software component.

In one embodiment, if all alternate code paths within the highest-levelsoftware component specified in the index also fail to circumvent theparticular error, the software system will determine which softwarecomponent invoked this highest-level software component by checking thecall stack. In this embodiment, it is possible to uniquely identifywhich component invoked the highest-level component specified in theindex because every exception handler in the software system records itscomponent name. Once the component is identified, the software systemwill then check if there is an index key entry that matches to the nameof this component, and will explore all alternate code paths specifiedby the corresponding index entry.

In one embodiment, the key-parts of the index key can be ordered in aparticular order. For example, the highest-ordered key-part cancorrespond to general components of the software system, and thelowest-ordered component can correspond to specific errors that areencountered in the software system. The step of removing a value fromthe set of search values is performed by removing the value associatedwith the lowest ordered key-part that has not yet been removed from thesearch set. In a different embodiment, the software system may keep aparameter indicating the maximum number of alternate code paths that canbe explored before the traversal of the index is terminated. Settingthis parameter to 0 will completely disable exploring of alternate codepaths.

4. Automatically Switching to Alterante Code Paths

In one embodiment, a particular operation is associated with aparameter, and a distinct value of the parameter is associated with eachcode path of the plurality of code paths for performing the particularoperation. The software system automatically switches from the currentexecution code path to a different code path by automatically changingthe value of the parameter from the value associated with the currentexecution code path to the value of the parameter associated with thenew different code path.

In an embodiment, a sequence is associated with the plurality of codepaths for executing a particular operation. During the switch-over, thesoftware system automatically switches the execution of the particularoperation from the current code path to the code path which immediatelyfollows it in the sequence. In a different embodiment, the sequence isbased on the probability that an error will be encountered when thesoftware system executes a code path from the plurality of code paths inresponse to a request to perform the particular operation. In yet adifferent embodiment, the sequence in which alternate code paths areselected is based on minimum deviation from the original execution codepath.

5. Timing and Scope of Switching to Alternate Code Paths

After the software system has determined that path-switching conditionshave been satisfied and has selected an alternative code path to performa particular operation, the software system changes the value of theparameter to the value associated with the new different code path. Inone embodiment, the parameter is a run-time parameter, and the settingof its value takes effect immediately. Any subsequent requests toperform the particular operation are performed by executing the newlyselected different code path. In another embodiment, the parameter isnot run-time, and the change of its value takes effect upon restartingthe software system. Whether to make the parameter a run-time parameterdepends on whether the correct execution of the particular operation canbe guaranteed without affecting other processes or threads that maystill be in the process of executing the original execution code path.

In one embodiment, when the software system automatically changes thevalue of a parameter to indicate a switch from the current executioncode path to a different code path of the plurality of code paths forperforming a particular operation, the change of code paths applies onlyto the process or thread for which path-switching condition have beensatisfied while executing the particular operation. In a differentembodiment, when the software system changes the value of a parameter tofacilitate a code path switch, the new code path selected as theexecution code path for the particular operation is established as thecode path for executing the particular operation for all subsequentrequests to perform the particular operation by all processes or threadsof the software system which can execute the particular operation.

In an embodiment, a software system comprises a plurality of instancesrunning in a cluster, that is, a plurality of virtually identical copiesof the software system executable code run on one or more hardwareplatforms. In this embodiment, the same code path is the currentexecution code path performed by each instance in the cluster inresponse to requests for performing a particular operation. When oneinstance of the software system establishes a different code path forperforming the particular operation, the new different code path isestablished as the execution code path for performing the operation onevery software system instance in the cluster.

6. Code Path Switching in Multi-Threaded Code with Shared State

Some software systems are designed to support a plurality of threads. Inexecution, the threads usually run concurrently and share the sameprocess state. The software system maintains a data structure in memorythat is shared by the plurality of threads. The shared memory datastructure may include one or more shared parameters. All the threads ofthe plurality of threads usually have the same rights to read and modifythe value of a shared parameter.

In multi-threaded software systems, although an error may be encounteredduring the execution of a particular operation by a first thread, theroot cause of the problem may be an incorrect execution of the same ordifferent operation in a second thread. The reason for this is that theerror encountered by the first thread may be because of a change to thevalue of a shared parameter by the second thread. Thus, in order for amulti-threaded software system to effectively circumvent software errorsencountered during the execution of the threads, it is necessary toexplore alternate code paths not only in the thread that reported theerror, but also in the thread that was the last modifier of the value ofa shared parameter.

In order to track the last modifier of the shared state, importantshared data structures include a last-modifier field that is set to thethread-id of the thread that modifies the value of a parameter includedin the data structure. When a first thread executing a particularoperation encounters an error due to an inconsistency in the shared datastructure, the first thread records, along with the rest of the errorinformation, the thread-id of a second thread (the thread that lastmodified the shared data structure). The software system will firstexplore alternate code paths for the particular operation executed bythe first thread. If the error is persistently encountered, the softwaresystem will then mark the second thread as suspect, and will investigatealternate code paths for the operation executed by the second thread.

In one embodiment, a software system is a multi-threaded system with aplurality of threads, which, in execution, share a data structure inmemory. A first thread of the software system performs a first operationby executing a first execution code path, of a plurality of distinctcode paths for performing the operation, in response to a request toperform the first operation. The software system further includes aplurality of distinct code paths for performing a second operation, anda second thread executes a second execution code path of the pluralityof code paths for the second operation in response to a request toperform the second operation. When the second thread modifies data inthe shared data structure, information about the second thread isautomatically recorded in the data structure. Based at least in part onthis recorded information, the software system automatically determineswhether it is warranted to switch from the second execution path for thesecond operation to another code path of the plurality of code paths forthe second operation, in response to the first thread encountering anerror while executing the first execution code path for performing thefirst operation. If the software system determines that switching thesecond execution path for the second operation is warranted, thesoftware system then selects an alternate code path from the pluralityof code paths for performing the second operation. In this example, thefirst and second operations may either be the same type of operation, ordifferent types of operation.

In an embodiment, the software system automatically determines whetherit is warranted to switch the second execution path for the secondoperation based on determining that the second thread modified data inthe shared data structure immediately before the first threadencountered the error. In a different embodiment, the software systemautomatically determines whether it is warranted to switch the secondexecution path for the second operation based on determining that thesecond thread modified the value of the parameter in the shared datastructure immediately before the first thread read the data.

In an embodiment, the software system first determines if all alternatecode paths of the plurality of code paths for the first operation havebeen explored. If the software system determines that all alternate codepaths for the first operation have previously been explored and thefirst thread still encounters an error, then the software systemdetermines that it is warranted to mark the second thread as suspect andto explore alternate code paths from the plurality of code paths for thesecond operation. The software system makes these determinations basedon error information which was recorded by the first thread and whichincludes an identifier of the second thread.

7. Selecting Alternate Code Paths During Regression Testing

In one embodiment, the software system is tested to determine whetherthere are any software errors. In this embodiment, when path-switchingconditions are satisfied, the software system randomly establishes adifferent code path of a plurality of distinct code paths for performinga particular operation. Such random selection of alternate code pathshas at least two benefits: (1) it ensures that all alternate code pathsare well tested and that the combinations of alternate code paths acrossdifferent software components work together, and (2) it improves theusefulness of old tests which have previously been written to test aparticular operation by providing different execution sequences and byincreasing the overall code coverage of a single test.

8. Functional Overview

FIG. 4 depicts a technique for automatically circumventing softwareerrors encountered by a software system.

In step 400, a plurality of distinct code paths for performing aparticular operation (herein named “OP”) is included in the softwaresystem. In step 405, one of this plurality of distinct code paths isestablished as the execution code path for operation OP, where thisexecution code path is executed by the software system in response torequests to perform operation OP. In step 410 the software systemreceives a request to perform operation OP. In step 415, the softwaresystem automatically determines whether conditions warranting theswitching of the current execution code path for operation OP aresatisfied. If in step 415 the software system determines that suchpath-switching conditions are not satisfied, then, in step 430, thesoftware system performs operation OP by executing the current executioncode path. If in step 415 the software system automatically determinesthat path-switching conditions are satisfied, then, in step 420, thesoftware system automatically establishes a different code path forperforming operation OP, where the different code path is selected fromthe plurality of distinct code paths for operation OP and is differentthan the current execution code path. In step 425, the software systemperforms operation OP by executing the different code path.

Hardware Overview

FIG. 5 is a block diagram that illustrates a computer system 500 uponwhich an embodiment of the invention may be implemented. Computer system500 includes a bus 502 or other communication mechanism forcommunicating information, and a processor 504 coupled with bus 502 forprocessing information. Computer system 500 also includes a main memory506, such as a random access memory (RAM) or other dynamic storagedevice, coupled to bus 502 for storing information and instructions tobe executed by processor 504. Main memory 506 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 504. Computersystem 500 further includes a read only memory (ROM) 508 or other staticstorage device coupled to bus 502 for storing static information andinstructions for processor 504. A storage device 510, such as a magneticdisk or optical disk, is provided and coupled to bus 502 for storinginformation and instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 514, including alphanumeric and other keys, is coupledto bus 502 for communicating information and command selections toprocessor 504. Another type of user input device is cursor control 516,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 504 and forcontrolling cursor movement on display 512. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

The invention is related to the use of computer system 500 forimplementing the techniques described herein. According to oneembodiment of the invention, those techniques are performed by computersystem 500 in response to processor 504 executing one or more sequencesof one or more instructions contained in main memory 506. Suchinstructions may be read into main memory 506 from anothermachine-readable medium, such as storage device 510. Execution of thesequences of instructions contained in main memory 506 causes processor504 to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

The term “machine-readable medium” as used herein refers to any mediumthat participates in providing data that causes a machine to operationin a specific fashion. In an embodiment implemented using computersystem 500, various machine-readable media are involved, for example, inproviding instructions to processor 504 for execution. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical or magnetic disks, such as storage device 510. Volatilemedia includes dynamic memory, such as main memory 506. Transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 502. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Common forms of machine-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punchcards, papertape, anyother physical medium with patterns of holes, a RAM, a PROM, and EPROM,a FLASH-EPROM, any other memory chip or cartridge, a carrier wave asdescribed hereinafter, or any other medium from which a computer canread.

Various forms of machine-readable media may be involved in carrying oneor more sequences of one or more instructions to processor 504 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 500 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 502. Bus 502 carries the data tomain memory 506, from which processor 504 retrieves and executes theinstructions. The instructions received by main memory 506 mayoptionally be stored on storage device 510 either before or afterexecution by processor 504.

Computer system 500 also includes a communication interface 518 coupledto bus 502. Communication interface 518 provides a two-way datacommunication coupling to a network link 520 that is connected to alocal network 522. For example, communication interface 518 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line.As another example, communication interface 518 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 518 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 520 typically provides data communication through one ormore networks to other data devices. For example, network link 520 mayprovide a connection through local network 522 to a host computer 524 orto data equipment operated by an Internet Service Provider (ISP) 526.ISP 526 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 528. Local network 522 and Internet 528 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 520and through communication interface 518, which carry the digital data toand from computer system 500, are exemplary forms of carrier wavestransporting the information.

Computer system 500 can send messages and receive data, includingprogram code, through the network(s), network link 520 and communicationinterface 518. In the Internet example, a server 530 might transmit arequested code for an application program through Internet 528, ISP 526,local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received,and/or stored in storage device 510, or other non-volatile storage forlater execution. In this manner, computer system 500 may obtainapplication code in the form of a carrier wave.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method for automatically keeping track of software errors, themethod comprising the computer-implemented steps of: detecting anoccurrence of an error that affects performance of an operation beingperformed by a database server; and in response to detecting theoccurrence, automatically recording error information about the error ina storage space within a database that is managed by the databaseserver; wherein the step of automatically recording the errorinformation is performed by executing one or more computer instructionsin a first code path of the database server; wherein the first code pathis a separate code path than a second code path of the database serverthat performs the operation whose performance is affected by the error.2. The method of claim 1, wherein the step of automatically recording isperformed by the database server.
 3. The method of claim 1, wherein thestep of automatically recording includes: determining whether thestorage space is filled up; and if the storage space is filled up, thenautomatically recording the error information over the oldest errorinformation existing in the storage space.
 4. The method of claim 1,wherein the first code path is a code path dedicated to automaticallyrecording the error information.
 5. The method of claim 1, wherein themethod further comprises the steps of: determining what type of erroroccurred; and selecting which type of error information to store in thestorage space based on the type of error that occurred.
 6. The method ofclaim 5, wherein the type of error information selected to be stored inthe storage space includes at least one of a value of a parametermaintained by the database server, a value of a parameter maintained byan operating system, and a value of a parameter associated with anetwork connection.
 7. The method of claim 1, further comprising thesteps of: receiving, from a user, a database command that requestsretrieval of the error information from the storage space; the databaseserver executing the database command to retrieve the error informationfrom the storage space; and returning the error information to the user.8. The method of claim 7, wherein the database command is a StructuredQuery Language (SQL) command.
 9. The method of claim 1, furthercomprising the step of receiving configuration information from a user,wherein the configuration information determines a location and a sizeof the storage space.
 10. The method of claim 1, wherein the step ofautomatically recording the error information comprises the steps of:determining whether errors have previously been detected; and recordingerror information only about errors that have not previously beendetected.
 11. The method of claim 10, wherein the step of determiningwhether the errors have previously been detected comprises determiningthat the errors have not been encountered within a preset interval oftime.
 12. A computer-readable storage medium storing one or moresequences of instructions for automatically keeping track of softwareerrors, which instructions, when executed by one or more processors,cause the one or more processors to perform the steps of: detecting anoccurrence of an error that affects performance of an operation beingperformed by a database server; and in response to detecting theoccurrence, automatically recording error information about the error ina storage space within a database that is managed by the databaseserver; wherein the step of automatically recording the errorinformation is performed by executing one or more computer instructionsin a first code path of the database server; wherein the first code pathis a separate code path than a second code path of the database serverthat performs the operation whose performance is affected by the error.13. The computer-readable storage medium of claim 12, wherein theinstructions that cause the one or more processors to perform the stepof automatically recording are included in a set of executableinstructions of the database server.
 14. The computer-readable storagemedium of claim 12, wherein the instructions that cause the one or moreprocessors to perform the step of automatically recording compriseinstructions which, when executed by the one or more processors, causethe one or more processors to perform the steps of: determining whetherthe storage space is filled up; and if the storage space is filled up,then automatically recording the error information over the oldest errorinformation existing in the storage space.
 15. The computer-readablestorage medium of claim 12, wherein the first code path is a code pathdedicated to automatically recording the error information.
 16. Thecomputer-readable storage medium of claim 12, wherein the one or moresequences of instructions further comprise instructions which, whenexecuted by the one or more processors, cause the one or more processorsto perform the steps of: determining what type of error occurred; andselecting which type of error information to store in the storage spacebased on the type of error that occurred.
 17. The computer-readablestorage medium of claim 16, wherein the type of error informationselected to be stored in the storage space includes at least one of avalue of a parameter maintained by the database server, a value of aparameter maintained by an operating system, and a value of a parameterassociated with a network connection.
 18. The computer-readable storagemedium of claim 12, wherein the one or more sequences of instructionsfurther comprise instructions which, when executed by the one or moreprocessors, cause the one or more processors to perform the steps of:receiving, from a user, a database command that requests retrieval ofthe error information from the storage space; the database serverexecuting the database command to retrieve the error information fromthe storage space; and returning the error information to the user. 19.The computer-readable storage medium of claim 18, wherein the databasecommand is a Structured Query Language (SQL) command.
 20. Thecomputer-readable storage medium of claim 12, wherein the one or moresequences of instructions further comprise instructions which, whenexecuted by the one or more processors, cause the one or more processorsto perform the step of receiving configuration information from a user,wherein the configuration information determines a location and a sizeof the storage space.
 21. The computer-readable storage medium of claim12, wherein the instructions that cause the one or more processors toperform the step of automatically recording the error informationcomprise instructions which, when executed by the one or moreprocessors, cause the one or more processors to perform the steps of:determining whether errors have previously been detected; and recordingerror information only about errors that have not previously beendetected.
 22. The computer-readable storage medium of claim 21, whereinthe instructions that cause the one or more processors to perform thestep of determining whether the errors have previously been detectedcomprise instructions which, when executed by the one or moreprocessors, cause the one or more processors to perform the step ofdetermining that the errors have not been encountered within a presetinterval of time.